Facial Expression Analysis for Human Computer Interaction

This thesis describes a research project aimed at detecting emotions based on facial expression analysis. The context of this project is the I.N.E.S. system which is an intelligent tutoring system. It provides a simulation environment for nursing students to practice administering medicine by subcutaneous injection. Componential Appraisal Theory is used to describe facial expressions in terms of Stimulus Evaluation Checks and their outcomes. Also the situations in which these facial expressions occur can be described in terms of Stimulus Evaluation Checks. Video data of students working with the I.N.E.S. system has been collected and the facial expressions by the students have been analyzed. Analysis shows insufficient amount of data is collected to perform a statistical analysis. Nevertheless, this labeling of facial expressions and situations can be used for designing a system that enables a tutor to adjust its learning strategies based on the facial expressions it detects. Further analysis also shows in some situations, some of the elements in the labeling have a stronger influence than the other elements. The history of a student, the previous situations and the facial expressions shown by the student are also very important for determining the learning strategy to be used. A feature based approach that uses colors and edges in images is used to recognize facial expressions from a video stream. A skin color filter is used to detect the region of the face in an image. The eyes and nose are found by searching in HSV space for dark spots in the face. A rule based system is used to recognize these spots as the pupils and the nostrils. Another color filter is used to find the mouth region. This works because the lips contain stronger red values than the surrounding skin. After the mouth is found, a second order Sobel edge detector is applied to find the shape of the mouth. The same edge detector is also applied to the region above the eyes to find the eyebrows. Head pose is detected by looking at the distances in a triangle formed by the two eyes and the nose. Comparison with known distances of a person’s face is used to correct head pose. The detected feature points are used in a state machine to determine the facial expression. The mouth can be classified to be in the state; smile, neutral or pull down mouth corners. The eyebrows can be classified to be in the state; raise, neutral or frown. The system is tested on a set of labeled videos of different people showing these facial expressions. The system scored 78.5% correct classification on the mouth and 57.5% on the eyebrows.